Collection Processing with Constraints, Monads, and Folds

We propose an intermediate form based on monad-algebra comprehensions (to represent queries), folds (to represent computation), and setoids over polynomial datatypes (to represent data), suitable for use in collection processing. Such an intermediate form captures, in a uniform way, large fragments of many recent largescale collection processing languages such as MapReduce, PIG, DryadLINQ, and Data Parallel Haskell, and admits optimization techniques from both programming language theory and relational database theory. We show how to solve four key problems inherent in the naive approach by drawing together recent work from both communities. First, we show how fold fusion can be extended, in an arguably complete way, to exploit the monadic structure of queries. Second, we show how monad comprehensions can be extended to monad-algebra comprehensions, so as to express aggregation and thereby enable comprehension-based optimizations for a wide class of queries. Third, we show how to embed a particular syntactic class of constraints called embedded dependencies into our intermediate form and show how such constraints can be used, for example, to minimize the number of bind operations in a monad comprehension a process traditionally known as semantic optimization. Finally, we show how to emit proof obligations from our language, so as to ensure that each program is sound with respect to required axioms (such as the monad laws) as well as user-provided invariants (such as ensuring all sets are represented by lists without duplicates).